For the classification of high-resolution remote sensing images, inspired by human vision system which extracts summary statistics information for scene perception, a feature extraction method based on summary features was proposed. In the method average orientation information and visual clutter were extracted and combined to form a representation based on summary statistics, in which average orientation information was summarized by using Gabor filters and visual clutter was measured based on visual crowding.The experimental results on the classification of 21 classes of remote sensing image set reveal that the classification accuracy of the proposed method is 6.5% higher than Gist and 3.22% higher than Bag-Of-Words (BOW), when the number of training images and testing images are both 50. It also has lower calculation burden. While compared with Gist, the proposed method doesn't need any human intervention.